Detecting user interactions with a computing system of a vehicle

ABSTRACT

A computing system of a vehicle for controlling a vehicle or systems within a vehicle. The computing system comprises a presence-sensitive panel within the vehicle, an infrared camera configured to capture images in a three-dimensional space within the vehicle, at least one processor; and at least one storage device that stores instructions. When the instructions are executed, they case the at least one processor to: receive, from the presence-sensitive panel, a first indication of input, receive, from the infrared camera, a second indication of input, and determine, based on at least one of the first indication of input or the second indication of input, an operation to be performed.

BACKGROUND

Vehicles, such as automobiles, motorcycles, aircraft, and watercraft,may include one or more computing systems for performing functions andproviding occupants of the vehicles with information, entertainment,assistance, or environmental control. For instance, an automobile mayinclude an entertainment system for playing music, videos, or othercontent, a navigation system for providing information and navigationalassistance, a temperature control system for heating or cooling thein-vehicle cabin, a control system for adjusting various components orfeatures of the car, such as a sun roof or window shades, or an“infotainment system” that performs some or all of these aforesaidfunctions.

In some cases, a computing system within a vehicle may require, foreffective operation, input from the driver or other occupant of thevehicle. For example, a computing system within a vehicle may requireconfirmation of a particular operation, or may require more specificinformation that might normally be expressed as voice input or as astring of characters. At times, a computing system within a vehicle mayrequire input from the driver when the driver is unable to provide hisor her full attention, such as when the driver's primary attention is onoperating the vehicle.

SUMMARY

In one example, a computing system of a vehicle comprises apresence-sensitive panel within the vehicle, an infrared cameraconfigured to capture images in a three-dimensional space within thevehicle, at least one processor, and at least one storage device. Thestorage device stores instructions that, when executed, cause the atleast one processor to: receive, from the presence-sensitive panel, afirst indication of input, receive, from the infrared camera, a secondindication of input, and determine, based on at least one of the firstindication of input or the second indication of input, an operation tobe performed.

In another example, a method comprises receiving, by a computing systemof a vehicle, from a presence-sensitive panel positioned within thevehicle, a first indication of input, receiving, by the computingsystem, from an infrared camera configured to capture images in athree-dimensional space within the vehicle, a second indication ofinput, and determining, based on at least one of the first indication ofinput or the second indication of input, an operation to be performed.

In another example, a computer-readable storage medium comprisesinstructions. The instructions, when executed by a computing system of avehicle, cause at least one processor of the computing system to:receive, from a presence-sensitive panel positioned within the vehicle,a first indication of input, receive, from an infrared camera configuredto capture images in a three-dimensional space within the vehicle, asecond indication of input, and determine, based on at least one of thefirst indication of input or the second indication of input, anoperation to be performed.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a conceptual diagram illustrating an interior of a vehiclethat includes an example vehicle computing system enabling an occupantof the vehicle to control systems within the vehicle through multipletypes of input.

FIG. 2 is a block diagram illustrating an example vehicle computingsystem that implements techniques for controlling vehicle systems withina vehicle in accordance with one or more aspects of the presentdisclosure.

FIG. 3A is a conceptual diagram illustrating a side view of an interiorof a vehicle, including an example vehicle computing system capable ofdetecting touch input in accordance with one or more aspects of thepresent disclosure.

FIG. 3B is a conceptual diagram illustrating a top view of an examplepresence-sensitive panel.

FIG. 3C is a conceptual diagram illustrating a perspective view from therear of the interior of a vehicle, including an example vehiclecomputing system capable of detecting touch input in accordance with oneor more aspects of the present disclosure.

FIG. 3D is a conceptual diagram illustrating a perspective view from therear of the interior of a vehicle, including an example vehiclecomputing system capable of providing feedback for touch input inaccordance with one or more aspects of the present disclosure.

FIG. 4A is a conceptual diagram illustrating a side view of an interiorof a vehicle, including an example vehicle computing system capable ofdetecting gesture input in an example field of view in accordance withone or more aspects of the present disclosure.

FIG. 4B is a conceptual diagram illustrating a top view of an examplepresence-sensitive panel and an example camera.

FIG. 4C is a conceptual diagram illustrating a perspective view from therear of the interior of a vehicle, including an example vehiclecomputing system capable of detecting gesture input in accordance withone or more aspects of the present disclosure.

FIG. 4D is a conceptual diagram illustrating a perspective view from therear of the interior of a vehicle, including an example vehiclecomputing system capable of providing feedback for gesture input inaccordance with one or more aspects of the present disclosure.

FIG. 4E is a conceptual diagram illustrating a perspective view from therear of the interior of a vehicle, including an example vehiclecomputing system capable of detecting another type of gesture input inaccordance with one or more aspects of the present disclosure.

FIG. 5 is a flow diagram illustrating example operations of an examplevehicle computing system that implements techniques for recognizingmultiple types of input, in accordance with one or more aspects of thepresent disclosure.

FIG. 6 is a flow diagram illustrating example operations of an examplevehicle computing system that implements techniques for recognizingmultiple types of input, in accordance with one or more aspects of thepresent disclosure.

DETAILED DESCRIPTION

This disclosure is directed to techniques for enabling a single,multi-tiered input system to detect both fine-grained and course-graineduser inputs for controlling one or more computing systems within avehicle (also referred to herein as “vehicle computing systems”). Forsome controls, a vehicle system may require a more detailed or so-calledfine-grained user input, such as when a vehicle computing systemrequests user input of a physical address used by a navigationapplication executing at the vehicle computing system or an emailaddress used by a communication application running on the vehiclecomputing system. For other controls, a vehicle system may require aless detailed or so-called course-grained user input, such as when avehicle computing system requests user input to confirm an operation orto control a cabin thermostat. And still for other controls, a vehiclesystem may require a combination of fine and course-grained inputs.

The example multi-tiered input system may provide a single userinterface from which a user can easily switch between providingfine-grained and course-grained inputs. The example multi-tiered inputsystem may rely on a touch panel configured to detect fine-grained userinput from an occupant of the vehicle (e.g., primarily astwo-dimensional gesture inputs made at or near an input surface of thetouch panel) and using a camera positioned near the touch panel, theexample multi-tiered input system may also be configured to detectcourse-grained user input (e.g., primarily as three-dimensional gestureinputs made in the air above the input surface of the touch panel).

The example multi-tiered input system may enable a vehicle computingsystem to detect both fine-grained input and course-grained input from asingle system that is within easy reach of a vehicle occupant and mayprovide the vehicle occupant a way to quickly and easily switch betweenproviding the appropriate input for a given situation. For example, evenwhen the driver is operating the vehicle, the driver can interact with asingle input system that easily switches between detecting fine-grainedand course-grained inputs without requiring the driver to switch orchange his or her input focus. Accordingly, a vehicle computing systemthat includes the example multi-tiered input system may provide aparticularly natural or easy user interface from which the driver (orother vehicle occupant) can use to provide input to one or more systemswithin a vehicle.

FIG. 1 is a conceptual diagram illustrating an interior of a vehiclethat includes an example vehicle computing system enabling an occupantof the vehicle to control systems within the vehicle through multipletypes of input. FIG. 1 shows a cross-sectional view of a vehicleinterior in addition to components of vehicle computing system 100.Vehicle computing system 100 is configured to detect and process userinput, including both detailed or fine-grade input in a two-dimensionalplane as well as less detailed or more course input in athree-dimensional space, in accordance with one or more aspects of thepresent disclosure.

The vehicle illustrated in FIG. 1 may be an automobile, but aspects ofthe present disclosure may also be applicable to other types ofvehicles, including trucks, motorcycles, aircraft, watercraft, trains,or other vehicles. In FIG. 1, a driver may normally occupy seat 152.Seat 152 of the automobile may be positioned directly behind steeringwheel 154 of the vehicle such that an occupant of seat 152 mayphysically control steering wheel 154. The seat 152 is positioned withinthe vehicle illustrated in FIG. 1 under roof 158. Steering wheel 154 mayprotrude from dashboard 156. At least one front passenger seat may belaterally positioned adjacent to seat 152. Other passenger seats may bepositioned behind seat 152 or in front of seat 152.

Also shown in FIG. 1 is a collection of devices, components, and modulesthat may each be included in vehicle computing system 100. Vehiclecomputing system 100 includes, but is not limited to, presence-sensitivepanel 102 and camera 104, as well as display 112 and control unit 106.One or more components of vehicle computing system 100, such aspresence-sensitive panel 102 and camera 104 may be directly andphysically accessible to occupants seated in the front driver and frontpassenger seats of the automobile, and may be located within, near, oron center console 101. Such components may be within easy reach of suchoccupants, and may also or alternatively be positioned in anotherpassenger area of the vehicle, such as a back seat. In some examples, acomponent may be within easy reach if a vehicle occupant does not needto change positions in his or her seat in order to reach the componentwith an outstretched arm. Stated another way, for many drivers, forexample, the usual positions of the steering wheel, stick shift, andcenter console may be considered within easy reach of the driver. Asfurther described below, presence-sensitive panel 102 and camera 104 mayfunction as input devices for vehicle computing system 100. In someexamples, one or more components of vehicle computing system 100 thatmight not necessarily require physical access by occupants of thevehicle (such as, in some examples, display 112 and control unit 106),may be positioned in or on or integrated into dashboard 156. Suchcomponents may be integrated as part of an automobile dashboard and/orconsole facing or near the occupants of the vehicle. As furtherdescribed in this disclosure, vehicle computing system 100 may includedisplay 112 that may output a graphical user interface.

As described and illustrated, some or all of vehicle computing system100 may be housed within dashboard 156, which may in some examples beconstructed of plastic, vinyl, rubber, aluminum, steel, or any othersuitable material. Control unit 106 may be housed within housing 105,which may also be constructed of plastic, vinyl, rubber, aluminum,steel, or any other suitable material. In some examples, housing 105 mayalso be a rigid case that encloses and otherwise protects one or moreelectrical components that provide functionality for vehicle computingsystem 100. In some examples, housing 105 may be affixed, mounted orotherwise integrated with the automobile dashboard or console.

Control unit 106 may provide an operating environment or platform forone or one more modules, such as a combination of hardware, firmware,and software, as further illustrated in FIG. 2. For instance, controlunit 106 may include one or more processors and storage devices that mayexecute instructions and store data of one or more modules. Control unit106 may also be operably coupled to one or more other software and/orhardware components, including presence-sensitive panel 102, camera 104,and display 112 to control, configure, and/or communicate informationwith the components, to name only a few example operations.

Display 112 may function as an output device, such as a display device,using any one or more of a liquid crystal display (LCD), dot matrixdisplay, light emitting diode (LED) display, organic light-emittingdiode (OLED) display, e-ink, or similar monochrome or color displaycapable of outputting visible information to a user or vehicle occupant.In some examples, display 112 may also function as an input device, sothat it serves as both an input and output device. In such examples,display 112 may include an integrated presence-sensitive input deviceand a display device. For instance, display 112 may function as apresence-sensitive input device using a presence-sensitive screen, suchas a resistive touchscreen, a surface acoustic wave touchscreen, acapacitive touchscreen, a projective capacitance touchscreen, apressure-sensitive screen, an acoustic pulse recognition touchscreen, oranother presence-sensitive screen technology. Based on user input,display 112 may present output to a user. For instance, display 112 maypresent various user interfaces of applications (e.g., a navigationapplication) executing at vehicle computing system 100. An occupant ofthe vehicle, such as a driver, may provide user input to interact withone or more of such applications.

Vehicle computing system 100 may operate to assist, inform, entertain,or perform other tasks that require user interactions with occupants ofa vehicle. Vehicle computing system 100 may be referred to as a vehiclehead unit, an infotainment system, or a subcomponent thereof. Forexample, vehicle computing system 100 may include one or moreapplication modules 110 that perform functions or process information,on behalf of one or more occupants of the vehicle. For instance, vehiclecomputing system 100 may provide a navigation service that providesdirections to destinations. Vehicle computing system 100 may alsoprovide an information retrieval service that provides information inresponse to queries and/or as preemptive assistance or recommendations.Vehicle computing system 100 may also provide vehicle data about thevehicle, or multimedia such as audio or video. Mentioned are only a fewexamples of the functionality that may be provided by vehicle computingsystem 100, and vehicle computing system 100 may provide many additionalcapabilities. In this and other ways, vehicle computing system 100 mayimprove the driving or riding experience for one or more occupants ofthe vehicle.

In some examples, vehicle computing system 100 may be controlled throughinput detected by presence-sensitive panel 102, through input detectedby camera 104, and/or through input detected by a combination ofpresence-sensitive panel 102 and camera 104. Vehicle computing system100 may also be controlled through input detected by one or moreadditional input devices (e.g., microphones, physical buttons orswitches, or other types of input devices).

Presence-sensitive panel 102 may, in some examples, function simply asan input device for touch input, provided by user input that may occurdirectly and physically at presence-sensitive panel 102. For instance,presence-sensitive panel 102 may function as a multi-touchpresence-sensitive input device using a presence-sensitive device, suchas a resistive touchscreen or touch panel, a surface acoustic wavetouchscreen or touch panel, a capacitive touchscreen or touch panel, aprojective capacitance touchscreen or touch panel, a pressure-sensitivescreen or touch panel, an acoustic pulse recognition touchscreen ortouch panel, or another presence-sensitive screen or touch paneltechnology. In some examples, presence-sensitive panel 102 may detect anobject at and/or near, or within range of the presence-sensitivecomponent(s) associated with presence-sensitive panel 102. As oneexample range, presence-sensitive panel 102 may detect an object, suchas a finger or stylus that is within 2 cm or less of presence-sensitivepanel 102. Presence-sensitive panel 102 may determine a location (e.g.,an (x,y) coordinate) of the presence-sensitive input device at which theobject was detected. In another example range, presence-sensitive panel102 may detect an object 6 inches or less from presence-sensitive panel102; other ranges are also possible. In some examples, input detected bypresence-sensitive panel 102 at ranges greater than 2 inches may be usedas a depth signal in conjunction with input detected by camera 104.Presence-sensitive panel 102 may detect a user's finger, stylus, orsimilar using capacitive, inductive, and/or optical recognitiontechniques.

In the example illustrated in FIG. 1, presence-sensitive panel 102 maybe positioned in center console 101 above camera 104, and center console101 may be transparent to camera 104 so that camera 104 may captureimages directly above presence-sensitive panel 102 even thoughpresence-sensitive panel 102 physically obscures the lens orfield-of-view of camera 104. For example, camera 104 may be an infraredcamera that captures images by receiving infrared light andpresence-sensitive panel 102 may be transparent to infrared light suchthat camera 104 is able to receive the infrared light originatingbetween the roof 158 and presence-sensitive panel 102. In otherexamples, camera 104 might not be positioned directly underpresence-sensitive panel 102, and camera 104 may be positioned elsewherewithin the vehicle.

In some examples, presence-sensitive panel 102 may function as both aninput device and as an output device. In such examples,presence-sensitive panel 102 may include an integratedpresence-sensitive input device and a display device, and could be anyone or more of a liquid crystal display (LCD), dot matrix display, lightemitting diode (LED) display, organic light-emitting diode (OLED)display, e-ink, or similar monochrome or color display capable ofoutputting visible information to a user or vehicle occupant. In otherexamples where presence-sensitive panel 102 includes both input deviceand output device functionality, presence-sensitive panel 102 may beimplemented by two separate components: a presence-sensitive inputdevice for receiving input and a display device for providing output. Inexamples where presence-sensitive panel 102 includes both input deviceand output device functionality, presence-sensitive panel 102 may stillbe positioned in center console 101 above camera 104, and center console101 may still be transparent to camera 104 so that camera 104 maycapture images directly above presence-sensitive panel 102, even ifpositioned under presence-sensitive panel 102.

Camera 104 may be one or more of any appropriate type of imageacquisition device, such as a camera or charge-coupled device. In someexamples, camera 104 may be one or more infrared cameras with a highfield-of-view and shallow depth of focus, and may be a backlit infraredcamera oriented to point generally upward within the vehicle, havingfield-of-view 160. In other examples, camera 104 may be or may furtherinclude one or more other types of cameras or image sensors, which mayinclude one or more other infrared cameras, thermographic cameras,thermal imaging cameras, light-sensitive cameras, range sensors,tomography devices, radar devices, or ultrasonic cameras. In someexamples, camera 104 may be any image capture device appropriate forapplication of computer vision techniques. Depending on the type ofsensors or cameras used, the resulting image may include two-dimensionalimages, three-dimensional volumes, or an image sequence. Pixel valuestypically correspond to light intensity in one or more spectral bands,but might also be related to various physical measures, such as depth,absorption or reflectance of sonic or electromagnetic waves, or nuclearmagnetic resonance. Field-of-view 160 shown in FIG. 1 is illustrative,and other dimensions, shapes, or volumes of field-of-view 160 arepossible.

In accordance with one or more aspects of the present disclosure, thecombination of presence-sensitive panel 102 and camera 104 may serve asa combined “multi-tiered” input device that may be able to detect avariety of types of input, of varying degrees of detail. For instance,presence-sensitive panel 102 may be configured for detecting touch inputdirectly at or physically near presence-sensitive panel 102 (e.g., witha finger or a stylus pen). Presence-sensitive panel 102 may detect oneor more tap and/or non-tap gestures, continuous gestures, or any othertouch gestures (e.g., multi-touch gestures) and vehicle computing system100 may determine, based on the input detected by presence-sensitivepanel 102, one or more symbols, gestures, or user commands correspondingto the input detected by presence-sensitive panel 102. Such a symbol,gesture, or command may be relatively detailed, and may include ahand-drawn number, character, or a string of characters determined fromthe detected input, and therefore presence-sensitive panel 102 may beused to capture what may be considered relatively detailed input orfine-grade input.

Camera 104 may be configured to capture movements of an occupant of thevehicle, such as a driver, as the occupant moves an arm, wrist, hand,stylus, and/or fingers as he or she gestures in, for example,field-of-view 160. Camera 104 may detect one or more images of a hand orarm signal, hand or arm movement, or gesture, and vehicle computingsystem 100 may be configured to determine, based on the input detectedby camera 104, a user command or signal corresponding to the detectedinput. Vehicle computing system 100 may be configured to recognize avariety of user gestures or user movements. For example, input detectedby camera 104 may be interpreted by vehicle computing system 100 as agesture requesting an increase (or decrease) of an in-cabin temperaturesetting. In another example, input detected by camera 104 may beinterpreted by vehicle computing system 100 as a gesture requesting anincrease (or decrease) of sound system volume. In other examples, inputdetected by camera 104 may be interpreted by vehicle computing system100 as a user gesture confirming or declining a particular operation. Auser command corresponding to the input detected by camera 104 may berelatively less detailed, and may correspond to simply a hand swipegesture, and therefore camera 104 may be used to capture what may beconsidered relatively less detailed input or more course input, at leastcompared to some input that may be detected by presence-sensitive panel102.

Vehicle computing system 100 may use computer vision techniques todetermine one or more user commands from the input detected by camera104. In the example shown in FIG. 1, camera 104 may be pointing directlyupwards, towards roof 158 of the vehicle, so in some examples,background subtraction and thresholding may be used to detect motionwithin field-of-view 160 (e.g., motion of a vehicle occupant's hand orarm). In such an example, vehicle computing system 100 may calculate theconvex hull and calculate convexity defects and estimate, for example,the position of the vehicle occupant's hand in three dimensional spacegenerally above presence-sensitive panel 102 and camera 104 (e.g.,within field-of-view 160). Vehicle computing system 100 mayalternatively or additionally determine gestures or movements infield-of-view 160 through alternative techniques.

As described above, vehicle computing system 100 may include userinterface (UI) module 108 and application modules 110. UI module 108 andapplication modules 110 may perform operations described herein usingsoftware, hardware, firmware, or a mixture of both hardware, software,and firmware residing in and executing by vehicle computing system 100or at one or more other remote computing devices. As such, UI module 108and application modules 110 may be implemented as hardware, software,and/or a combination of hardware and software. Vehicle computing system100 may execute UI module 108, application modules 110, or one or moreother modules as or within a virtual machine executing on underlyinghardware. UI module 108 and application modules 110 may be implementedin various ways. For example, UI module 108 and application modules 110may be implemented as a downloadable or pre-installed application or“app.” In another example, UI module 108 and application modules 110 maybe implemented as part of an operating system of vehicle computingsystem 100.

Application modules 110 may include functionality to perform any varietyof operations on vehicle computing system 100. For instance, applicationmodules 110 may include a navigation application, weather application, aphone dialer application, an information retrieval application, amultimedia application, a vehicle information application, an emailapplication, a text messing application, instant messaging application,social networking application, weather application, stock marketapplication, emergency alert application, sports application, to nameonly a few examples. In general, vehicle computing system 100, whetherthrough application modules 110 or otherwise, may be configured toperform operations including those relating to climate control systems(e.g., heating and air conditioning), audio or infotainment systems,seat, window, sunshade, or windshield wipers, cruise control, in-cabindisplay system, steering wheel controls, headrest, arm rest, side orrear view mirrors, collision sensors. Such operations may be controlledby one or more application modules 110, or may be controlled by othersystems within the vehicle. In some examples, such operations may belimited to non-safety features of the vehicle. In other examples, suchoperations may encompass one or more features of the vehicle that may beconsidered safety-related (e.g., turning on a turn-signal, adjusting amirror, adjusting or fastening/disconnecting a seat belt, adjustingcruise control features, accelerating, braking).

Although shown as operable within control unit 106 of vehicle computingsystem 100, one or more of application modules 110 may be operable by aremote computing device that is communicatively coupled to vehiclecomputing system 100. In such examples, an application module executingat a remote computing device may cause the remote computing device tosend the content and intent information using any suitable form of datacommunication (e.g., wired or wireless network, short-range wirelesscommunication such as Near Field Communication or Bluetooth, etc.). Insome examples, a remote computing device may be a computing device thatis separate from a computing device included in vehicle computing system100. For instance, the remote computing device may be operativelycoupled to vehicle computing system 100 by a network. An example of aremote computing device may include, but is not limited to a server,smartphone, tablet computing device, smart watch, and desktop computer.In some examples, a remote computing device may or may not be anintegrated component of vehicle computing system 100.

UI module 108 of vehicle computing system 100 may receive frompresence-sensitive panel 102 one or more indications of user inputdetected at presence-sensitive panel 102. Generally, each timepresence-sensitive panel 102 detects user input at a particular locationof presence-sensitive panel 102, UI module 108 may receive an indicationof user input or information about the user input frompresence-sensitive panel 102. UI module 108 may assemble the informationreceived from presence-sensitive panel 102 into a set of one or moreevents, such as a sequence of one or more touch events or gestureevents. Each gesture event in the sequence may include data orcomponents that represent parameters (e.g., when, where, originatingdirection) characterizing a presence and/or movement of input atpresence-sensitive panel 102. Each gesture event in the sequence mayinclude a location component corresponding to a location ofpresence-sensitive panel 102, a time component related to whenpresence-sensitive panel 102 detected user input at the location, and/oran action component related to whether the gesture event corresponds toa lift up or a push down at the location.

UI module 108 may determine one or more characteristics of the userinput based on the sequence of gesture events and include informationabout these one or more characteristics within each gesture event in thesequence of gesture events. For example, UI module 108 may determine astart location of the user input, an end location of the user input, adensity of a portion of the user input, a speed of a portion of the userinput, a direction of a portion of the user input, and a curvature of aportion of the user input. UI module 108 may transmit indications ofuser input from presence-sensitive panel 102 to other modules, such asapplication modules 110. UI module 108 may determine one or more single-or multi-touch gestures provided by a user. UI module 108 may also actas an intermediary between various components of vehicle computingsystem 100 to make determinations based on input detected bypresence-sensitive panel 102 and generate output presented by display112. For instance, UI module 108 may receive data from one or moreapplication modules 110 and cause display 112 to output content, such asa graphical user interface, for display.

UI module 108 of vehicle computing system 100 may also receive fromcamera 104 one or more indications of user input detected by camera 104.Generally, each time camera 104 detects a user gesture or movement infield-of-view 160, UI module 108 may receive an indication of user inputor information about the user input from camera 104. UI module 108 mayassemble the information received from camera 104 into a set of one ormore events, such as a sequence of movements or gesture events. Eachgesture event in the sequence may include data or components thatrepresents parameters (e.g., when, where in three dimensional space,originating direction, direction in three dimensional space, hand or armorientation or posture) characterizing a presence, gesture, and/ormovement captured by camera 104 within field-of-view 160. Each gestureevent in the sequence may include a location component corresponding toa three-dimensional location within field-of-view 160, a time componentrelated to when camera 104 detected user input within thethree-dimensional space, an action component related to what type ofgesture was made, and/or one or more images captured by camera 104.

UI module 108 may determine one or more characteristics of the userinput based on the sequence of gesture events and include informationabout these one or more characteristics within each gesture event in thesequence of gesture events. For example, UI module 108 may determine astart location of the user gesture, an end location of the user gesture,a direction in three dimensional space within field-of-view 160 of theuser gesture, a speed of the movement of the user gesture, a directionof at least a portion of the user gesture, and may also determine aninterpretation of the type gesture that corresponds to the sequence ofgesture events. UI module 108 may transmit indications of user inputfrom camera 104 to other modules, such as application modules 110. UImodule 108 may determine one or more gestures performed by a user. UImodule 108 may also act as an intermediary between various components ofvehicle computing system 100 to make determinations based on movementsor gestures detected by camera 104 and generate output presented bydisplay 112, and UI module 108 may receive data from one or moreapplication modules 110 and cause display 112 to output content, such asa graphical user interface, for display.

In the example of FIG. 1, application modules 110 may include anavigation application that may be controlled through user inputdetected by presence-sensitive panel 102 and camera 104. In such anexample, presence-sensitive panel 102 may detect one or more user inputsat locations of presence-sensitive panel 102. In response to detectingthe one or more user inputs at locations of presence-sensitive panel102, presence-sensitive panel 102 may output to UI module 108 anindication of user input detected by presence-sensitive panel 102. UImodule 108 may output to the navigation application (one of applicationmodules 110) information about the detected input. Responsive to theinformation about the input, the navigation application may determine astring of characters that corresponds to the input, and may use thatstring of characters as a destination address. The navigationapplication may present the determined address to the user byoutputting, to UI module 108, the string of characters as part of aprompt to confirm the address as the desired destination. UI module 108may format and display that prompt requesting confirmation of theaddress on display 112.

Camera 104 may detect one or more movements made by the driver withinfield-of-view 160, and in response to detecting the one or moremovements within field-of-view 160, camera 104 may output to UI module108 an indication of user input detected by camera 104. UI module 108may output to the navigation application information about the detectedinput. Responsive to the information about the detected input, thenavigation application may determine a command that corresponds to theinput, and may interpret such command as the driver's confirmation thatthe destination address presented at display 112 is the desireddestination address. In response, the navigation application may proceedto provide audio and/or visual navigation guidance to the desireddestination. The navigation application may cause display 112 to outputa graphical user interface for display that corresponds to a map. Thenavigation application may cause the graphical user interface to provideone or more directions to the specified destination. For instance, theone or more directions may be displayed by display 112 as turn-by-turndirections based on the vehicle's current location, a text list ofdirections, or a high-level map view with a progress indicator to thedestination. During the drive, further input may be detected from thedriver or other occupant of the vehicle. Such input may be provided atpresence-sensitive panel 102. Such input may also be detected by camera104 in response to gestures or movements made in field-of-view 160.While the driver is navigating the vehicle, gestures made infield-of-view 160 may be easier to perform than some input that may beprovided at presence-sensitive panel 102.

In some examples, input detected by presence-sensitive panel 102 orcamera 104 may be interpreted differently by vehicle computing system100 depending on the application being executed by vehicle computingsystem 100 or the context or state of vehicle computing system 100. Inother words, in the example above describing a navigation application,certain taps, gestures, movements, or other input may be interpreted inthe context of the application. Such taps, gestures, movements, or otherinput may be interpreted differently if detected while another one ofthe application modules 110 (e.g., an infotainment application) is beingexecuted or if the vehicle computing system 100 is in a different stateor context. In still other examples, some taps, gestures, movements orother inputs may be interpreted by vehicle computing system 100 in aglobal manner, such that whenever such input is detected, the input isinterpreted by vehicle computing system 100 the same way. For example, aparticular gesture that may correspond to lowering the volume of one ormore audio output devices, may lower the volume in most or allsituations.

Although presence-sensitive panel 102 and camera 104 are described inFIG. 1 in the context of operation within a vehicle, in other examples,such an input system may be used in other contexts. For example, apresence-sensitive panel and camera that operates in a manner similar tothat described in connection with FIG. 1 may be used in a desktopcomputing systems, mobile computing systems, laptops, mobile devices, ordistributed computing systems, to name a few examples.

Since both fine-grained input and course-grained input can be detectedby vehicle computing system 100, a vehicle occupant can provide inputthat is both safe and appropriate for a given situation. For example,when operating the vehicle, the driver might find it difficult tointeract with a touchscreen or a presence-sensitive display for a numberof reasons, including the lack of tactile feedback and the need toconcentrate on operating the vehicle. Yet when the driver is operatingthe vehicle, and may be unable to devote full attention elsewhere, thedriver may nevertheless be able to provide a course-grained gesture.Thus, in some situations, a course-grained gesture may be anappropriate, safe, convenient, and effective alternative to the driverfinding and interacting with a physical button, knob, or switch, or atouchscreen button, icon, or interface. Also, in some cases, simpleinput is all that may be called for, and a course-grained gesture may besufficient even if the driver is not operating the vehicle. Yet forcases in which fine-grained input is more appropriate, such as when thedriver or other vehicle occupant is able to devote further attention toproviding input, and/or where fine-grained or detailed input may berequired, the vehicle computing system 100 is capable of detecting suchinput. Accordingly, in accordance with one or more aspects of thepresent disclosure, vehicle computing system 100 may be operated in avehicle or driver-safe way through a full range of input types,including both low and high fidelity gestures, movements, taps, signals,and other types of input, as well as gestures, movements, taps, signals,and other types of input between those extremes.

Further, in some examples, the arrangement and/or placement ofpresence-sensitive panel 102 and camera 104 within the vehicle mayprovide an ergonomic and comfortable way for a driver (or other vehicleoccupant) to interact with vehicle computing system 100. Whilepresence-sensitive panel 102 and camera 104 may detect different typesof input, the positioning of presence-sensitive panel 102 and camera 104in accordance with one or more aspects of this disclosure may be suchthat input detected by presence-sensitive panel 102 may be perceived bya vehicle occupant to be a natural extension of input detected by camera104. Similarly, input detected by camera 104 may be perceived by avehicle occupant to be a natural extension of input detected bypresence-sensitive panel 102. In other words, such a system may providea particularly natural or easy user interface for a vehicle occupant touse. In some cases, a vehicle occupant may find interacting with vehiclecomputing system 100 to be relatively instinctive.

In some examples, vehicle computing system 100 or certain components ofvehicle computing system 100 may be chosen to enable the overall cost ofimplementing vehicle computing system 100 to remain relatively low. Itmay be possible to implement vehicle computing system 100 usingcomponents that are relatively inexpensive, at least in comparison toalternative systems that may involve components different than, or inaddition to, those described in accordance with one or more aspects ofthe present disclosure.

Throughout the disclosure, examples are described where a computingdevice and/or a computing system analyzes information (e.g., context,locations, speeds, search queries, etc.) associated with a computingdevice and a user of a computing device, only if the computing devicereceives permission from the user of the computing device to analyze theinformation. For example, in situations discussed below, before acomputing device or computing system can collect or may make use ofinformation associated with a user, the user may be provided with anopportunity to provide input to control whether programs or features ofthe computing device and/or computing system can collect and make use ofuser information (e.g., information about a user's current location,current speed, etc.), or to dictate whether and/or how to the deviceand/or system may receive content that may be relevant to the user. Inaddition, certain data may be treated in one or more ways before it isstored or used by the computing device and/or computing system, so thatpersonally-identifiable information is removed. For example, a user'sidentity may be treated so that no personally identifiable informationcan be determined about the user, or a user's geographic location may begeneralized where location information is obtained (such as to a city,ZIP code, or state level), so that a particular location of a usercannot be determined. Thus, the user may have control over howinformation is collected about the user and used by the computing deviceand computing system.

FIG. 2 is a block diagram illustrating an example vehicle computingsystem 100 that implements techniques for controlling vehicle systemswithin a vehicle in accordance with one or more aspects of the presentdisclosure. Computing device 200 of FIG. 2 is described below within thecontext of FIG. 1. FIG. 2 illustrates only one particular example ofcomputing device 200, and many other examples of computing device 200may be used in other instances and may include a subset of thecomponents included in example computing device 200 or may includeadditional components not shown in FIG. 2. In the example of FIG. 2,vehicle computing system 100 includes one or more computing devices 200.In some examples, computing device 200 may be affixed, mounted orotherwise integrated with vehicle computing system 100.

As shown in the example of FIG. 2, computing device 200 may includepresence-sensitive panel 102, camera 104, one or more input devices 210,power source 214, one or more processors 218, one or more storagedevices 220, one or more output devices 208, one or more sensors 212,and one or more communication units 216.

Control unit 106 may encompass processors 218 and storage devices 220.Storage devices 220 may include UI module 108 and application modules110. Communication channels 222 may interconnect one or more of thecomponents identified above or shown in FIG. 2 for inter-componentcommunications (physically, communicatively, and/or operatively). Insome examples, communication channels 222 may include a system bus, anetwork connection, one or more inter-process communication datastructures, or any other components for communicating data betweenhardware and/or software.

As shown in FIG. 2, control unit 106 may store and execute the data andinstructions of one or more applications, modules or other software.Although FIG. 2 illustrates control unit 106 as including one or moreprocessors 218 and one or more storage devices 220, control unit 106 mayinclude more or fewer components than shown in FIG. 2. For instance,control unit 106 may include one or more output devices, input devices,input/output ports or interface, sensors and/or communication units toname only a few examples. In other examples, control unit 106 may onlyinclude one or more processors. In most cases, control unit 106 mayprovide an operating environment for one or one more modules, such as UImodule 108 and application modules 110.

One or more processors 218 may implement functionality and/or executeinstructions within computing device 200. For example, processors 218 ofcomputing device 200 may receive and execute instructions stored bystorage devices 220 that provide the functionality of UI module 108 andapplication modules 110. These instructions executed by processors 218may cause computing device 200 to store and/or modify information withinstorage devices 220 during program execution. Processors 218 may executeinstructions of UI module 108 and application modules 110. That is, UImodule 108 and application modules 110 may be operable by processors 218to perform various functions described herein.

Presence-sensitive panel 102, camera 104, and/or one or more other inputdevices 210 of computing device 200 may receive input. Examples of inputinclude tactile, audio, kinetic, and optical input, gestures, movements,and images, to name only a few examples. In addition topresence-sensitive panel 102 and camera 104, input devices 210 ofcomputing device 200, in some examples, may include a mouse orequivalent device, keyboard, voice responsive system, video camera,buttons, control pad, microphone or any other type of device fordetecting input from a human or machine. In some examples, such an inputdevice may be a presence-sensitive input device, which may include apresence-sensitive screen or touch-sensitive screen to name only a fewexamples.

One or more output devices 208 of computing device 200 may generate,receive, or process output. Examples of output are tactile, audio, andvideo output. Output devices 208 of computing device 200, in oneexample, include a presence-sensitive screen, sound card, video graphicsadapter card, speaker, cathode ray tube (CRT) monitor, liquid crystaldisplay (LCD), or any other type of device for generating output to ahuman or machine. Output devices 208 may include display devices such ascathode ray tube (CRT) monitor, liquid crystal display (LCD),Light-Emitting Diode (LED) or any other type of device for generatingtactile, audio, and/or visual output.

In some examples, display 112 is an output device such as cathode raytube (CRT) monitor, liquid crystal display (LCD), Light-Emitting Diode(LED) or any other type of device for generating tactile, audio, and/orvisual output.

In other examples, display 112 may include both input device and outputdevice functionality. In such an example, display 112 may include apresence-sensitive input device, such as a presence-sensitive screen ortouch-sensitive screen. Display 112 may detect an object at and/or near,or within range of the presence-sensitive component(s) associated withdisplay 112. Display 112 may determine the particular location(s) on ornear the surface of display 112 that have been selected by a user'sfinger, stylus, or similar using capacitive, inductive, and/or opticalrecognition techniques. In some examples where display 112 includes bothinput device and output device functionality, display 112 may beimplemented by two separate components: a presence-sensitive inputdevice for receiving input and a display device for providing output.

One or more communication units 216 of computing device 200 maycommunicate with external devices by transmitting and/or receiving data.For example, computing device 200 may use communication units 216 totransmit and/or receive radio signals on a radio network such as acellular radio network. In some examples, communication units 216 maytransmit and/or receive satellite signals on a satellite network such asa Global Positioning System (GPS) network. Examples of communicationunits 216 include a network interface card (e.g. such as an Ethernetcard), an optical transceiver, a radio frequency transceiver, a GPSreceiver, or any other type of device that can send and/or receiveinformation. Other examples of communication units 216 may includeBluetooth®, GPS, 3G, 4G, and Wi-Fi® radios found in mobile devices aswell as Universal Serial Bus (USB) controllers and the like.

One or more storage devices 220 within computing device 200 may storeinformation for processing during operation of computing device 200. Insome examples, one or more storage devices 220 are temporary memories,meaning that a primary purpose of the one or more storage devices is notlong-term storage. Storage devices 220 on computing device 200 may beconfigured for short-term storage of information as volatile memory andtherefore not retain stored contents if deactivated. Examples ofvolatile memories include random access memories (RAM), dynamic randomaccess memories (DRAM), static random access memories (SRAM), and otherforms of volatile memories known in the art.

Storage devices 220, in some examples, also include one or morecomputer-readable storage media. Storage devices 220 may be configuredto store larger amounts of information than volatile memory. Storagedevices 220 may further be configured for long-term storage ofinformation as non-volatile memory space and retain information afteractivate/off cycles. Examples of non-volatile memories include magnetichard discs, optical discs, floppy discs, flash memories, or forms ofelectrically programmable memories (EPROM) or electrically erasable andprogrammable (EEPROM) memories. Storage devices 220 may store programinstructions and/or data associated with UI module 108 and/orapplication modules 110.

As shown in FIG. 2, computing device 200 may include one or more sensors212. Sensors 212 may include an accelerometer that generatesaccelerometer data. Accelerometer data may indicate an accelerationand/or a change in acceleration of computing device 200. Sensors 212 mayinclude a gyrometer that generates gyrometer data. Gyrometer data mayindicate a physical orientation and/or change in physical orientation ofcomputing device 200. In some examples, the orientation may be relativeto one or more reference points. Sensors 212 may include a magnetometerthat generates magnetometer data. Magnetometer data may indicate themagnetization of an object that is touching or in proximity to computingdevice 200. Magnetometer data may indicate the Earth's magnetic field,and in some examples, provide directional functionality of a compass.Sensors 212 may include an ambient light sensor that generates ambientlight data. The ambient light data may indicate an intensity of light towhich computing device 200 is exposed. Sensors 212 may include aproximity sensor that generates proximity data. Proximity data mayindicate whether an object is within proximity to computing device 200.In some examples, proximity data may indicate how close an object is tocomputing device 200. In some examples, sensors 212 may include a clockthat generates a date and time. The date and time may be a current dateand time. Sensors 212 may include temperature sensor that measuresambient temperature in proximity to sensors 212. The ambient temperaturemay indicate an intensity of temperature.

As shown in FIG. 2, computing device 200 may include a power source 214.In some examples, power source 214 may be a battery. Power source 214may provide power to one or more components of computing device 200.Examples of power source 214 may include, but are not necessarilylimited to, batteries having zinc-carbon, lead-acid, nickel cadmium(NiCd), nickel metal hydride (NiMH), lithium ion (Li-ion), and/orlithium ion polymer (Li-ion polymer) chemistries. In some examples,power source 214 may have a limited capacity (e.g., 1000-3000 mAh).

In operation, presence-sensitive panel 102 may detect one or more taps,gestures, and/or other user inputs at locations of presence-sensitivepanel 102. In response, presence-sensitive panel 102 may output to UImodule 108 an indication of input detected by presence-sensitive panel102. In some examples, UI module 108 may determine, based on theindication of input, information about the input. Such information may,for example, indicate one or more lines, characters, or shapescorresponding to the input. UI module 108 may output to one or moreapplication modules 110 information about the input. In response to theinformation about the input, one or more application modules 110 mayperform an operation. In some examples, one or more application modules110 may output to one or more output devices 208, display 112, or anyother component shown in FIG. 2, information about the input. In someexamples, one or more application modules 110 may cause UI module 108 toupdate a graphical user interface to include information about theinput, the operation, or an operation to be performed. Such a graphicaluser interface may be presented at display 112 or at another outputdevice shown in FIG. 2.

Similarly, camera 104 may detect input in the form of one or more imagesof movements or gestures made within field of view 160. In response,camera 104 may output to UI module 108 an indication of input detectedby camera 104. In some examples, UI module 108 may determine, based onthe indication of input, information about the input. Such informationmay, for example, indicate one or more positions, motions, movements,sequences of movements, gestures, or gesture events corresponding to theinput. UI module 108 may output to one or more application modules 110information about the input. In response to the information about theinput, one or more application modules 110 may perform an operation. Insome examples, one or more application modules 110 may output to one ormore output devices 208, display 112, or any other component shown inFIG. 2, information about the input. In some examples, one or moreapplication modules 110 may cause UI module 108 to update a graphicaluser interface to include information about the input, the operation, oran operation to be performed. Such a graphical user interface may bepresented at display 112 or at another output device shown in FIG. 2.

FIG. 3A through FIG. 3D illustrate one or more example operationsrelating to detecting input by presence-sensitive panel 102. Forpurposes of illustration, FIG. 3A through FIG. 3D are described withinthe context of vehicle computing system 100 of FIG. 1 and FIG. 2.

FIG. 3A is a conceptual diagram illustrating a side view of an interiorof a vehicle, including an example vehicle computing system 100 capableof detecting touch input in accordance with one or more aspects of thepresent disclosure. Seated on seat 152 is user 150. User 150 may be adriver, but user 150 could also be a passenger or other vehicleoccupant. Although in FIG. 3A user 150 is shown in a position that mayoften be considered a front seat (characterized, e.g., by steering wheel154 and dashboard 156), user 150 may be seated in another locationwithin the vehicle, including a back seat.

In the example of FIG. 3A, user 150 may navigate or operate the vehicle,may interact with one or more components of the vehicle, and/or mayprovide input at input devices 210 or presence-sensitive panel 102 orcamera 104. In FIG. 3A, user 150 is shown interacting withpresence-sensitive panel 102.

Presence-sensitive panel 102 may detect one or more taps, gestures,and/or other user inputs at locations of presence-sensitive panel 102.Such taps, gestures, or other inputs may be from one or more fingers ofuser 150, or may be from a stylus or another device controlled by user150. Such input may be on the surface of presence-sensitive panel 102,or within a threshold distance of the surface of presence-sensitivepanel 102. In the illustration of FIG. 3A, the threshold distance mayextend above presence-sensitive panel 102, towards roof 158. Thethreshold distance may extend to roof 158, or may extend an approximatedistance, such as two feet, which may be appropriate to capture a handraised by a passenger in a seated position to approximately shoulderheight. A further distance is possible, but a shorter distance mayreduce the potential for noisy signals, such as those created by shadowsmoving on the roof of the vehicle.

In response to detecting the one or more inputs at locations ofpresence-sensitive panel 102, presence-sensitive panel 102 may output toUI module 108 an indication of input detected by presence-sensitivepanel 102. In some examples, UI module 108 may determine, based on theindication of input, information about the input. Such information may,for example, indicate one or more lines, characters, or shapescorresponding to the input. UI module 108 may output to one or moreapplication modules 110 information about the input. In response to theinformation about the input, one or more application modules 110 maydetermine an operation corresponding to the input and/or perform anoperation. In some examples, and in response to the information aboutthe input, one or more application modules 110 may output to display 112information about the input, the operation, or an operation to beperformed.

FIG. 3B is a conceptual diagram illustrating a top view of an examplepresence-sensitive panel 102. In the example of FIG. 3B, camera 104 ismounted under presence-sensitive panel 102, and the position or locationof camera 104 in FIG. 3B is illustrated by a dotted line, so in the viewshown, camera 104 may be concealed. In some examples, presence-sensitivepanel 102 may be transparent, or in other examples, presence-sensitivepanel 102 may be at least partially transparent in some respects. Forexample, presence-sensitive panel 102 may be at least partiallytransparent to infrared light. Accordingly, in some examples, camera 104may be partially visible through presence-sensitive panel 102. In someexamples, presence-sensitive panel 102 may serve as both an input deviceand an output or display device. If presence-sensitive panel 102 iscapable of operating as an output device, presence-sensitive panel 102may present information or feedback in response to input detected, ormay present a graphical user interface.

In FIG. 3B, presence-sensitive panel 102 may detect input along thesurface of presence-sensitive panel 102, and computing device 200 maydetermine, based on the input, the coordinates, shape, and direction ofline 302 traced by, for example, a finger or stylus along the surface ofpresence-sensitive panel 102. In other examples, computing device 200may determine, based on the indication of input, one or more characterscorresponding to the input. In still other examples, computing device200 may determine, based on the indication of input, that the inputcorresponds to selection of a graphical user interface element presentedon presence-sensitive panel 102, such as user interface element 303.

In some examples, presence-sensitive panel 102 may serve as an inputdevice, but may also have full or partial display capability. Forexample, presence-sensitive panel 102 may detect input corresponding toa finger moved along line 302 on surface of presence-sensitive panel102, and may output to UI module 108 an indication of input. In responseto the indication of input, computing device 200 may cause a line orother representation of the input detected by presence-sensitive panel102 to be presented on presence-sensitive panel 102 at or near line 302.

In some examples, computing device 200 may cause presence-sensitivepanel 102 to present a graphical user interface, which may include oneor more user interface elements 303. In such examples,presence-sensitive panel 102 may detect inputs at or near a location onpresence-sensitive panel 102 where presence-sensitive panel 102 presentsuser interface element 303, and in response, computing device 200 maydetermine that the detected input corresponds to selection of userinterface element 303. Responsive to such a determination, computingdevice 200 may perform an operation.

In other examples, presence-sensitive panel 102 may serve as an inputdevice, without providing any display capability. In such an example,presence-sensitive panel 102 might not display any line or otherfeedback corresponding to input detected by presence-sensitive panel102. Computing device 200 may cause feedback or other informationresponsive to input detected at presence-sensitive panel 102 to bedisplayed elsewhere, such as at display 112, as further described below.

FIG. 3C is a conceptual diagram illustrating a perspective view from therear of the interior of a vehicle, including an example vehiclecomputing system 100 capable of detecting touch input in accordance withone or more aspects of the present disclosure. FIG. 3C illustratescenter console 101 with camera 104 positioned on or within centerconsole 101. Presence-sensitive panel 102 is positioned on top of camera104 in a substantially horizontal position or orientation along centerconsole 101. Presence-sensitive panel 102 may, in some examples, bewithin easy reach of the driver or other occupant(s) of the vehicle.Also shown in FIG. 3C is steering wheel 154, which may be on either sideof center console 101, or in another location. Display 112, which may bewithin view of the driver or other occupant(s) of the vehicle, maypresent a graphical user interface. In the example of FIG. 3C, display112 presents a graphical user interface that includes line 305, cursor306, prompt 308, and one or more user interface elements 304.

UI module 108 of computing device 200 may receive indications of taps,gestures, and/or other input at locations of presence-sensitive panel102. Computing device 200 may determine, based on the input, the shapeof a line drawn along the surface of presence-sensitive panel 102.Computing device 200 may cause display 112 to present line 305, whichmay correspond to the shape of the line drawn along the surface ofpresence-sensitive panel 102. In this way, computing device 200 mayprovide visual feedback in response to input detected atpresence-sensitive panel 102.

Computing device 200 may also or alternatively determine, based on theinput at presence-sensitive panel 102, one or more characterscorresponding to the input. In some examples, each time computing device200 determines a character, computing device 200 may format and outputfor display at display 112 the determined character(s). Computing device200 may determine a string of characters in this way, and may perform anoperation based on the string characters. The string of characters may,for example, correspond to a physical address that may be used in anavigation application. In some examples, computing device 200 mayformat and output for display at display 112 the string of characters.In some examples, computing device 200 may alternatively, or inaddition, format and output for display at presence-sensitive panel 102one or more of such characters, for example, as one or more charactersare determined.

Cursor 306 may be presented within the graphical user interfacepresented at display 112. In some examples, cursor 306 may act as aplaceholder marking or corresponding to the most recent input atpresence-sensitive panel 102. For instance, computing device 200 maycause cursor 306 to be displayed at a location on display 112 thatcorresponds to the location at presence-sensitive panel 102 where thelast input was detected at presence-sensitive panel 102. In other words,if presence-sensitive panel 102 detects input corresponding to a linedrawn on the surface of presence-sensitive panel 102, computing device200 may position cursor 306 on display 112 at a location correspondingto where the last input was detected on presence-sensitive panel 102. Ifcomputing device 200 determines that the last input onpresence-sensitive panel 102 was in the upper right hand corner ofpresence-sensitive panel 102, computing device 200 may position cursor306 in the upper right hand corner of display 112. Similarly, ifcomputing device 200 determines that the most recent input atpresence-sensitive panel 102 was at the very center ofpresence-sensitive panel 102, computing device 200 may present cursor306 at the very center of the graphical user interface presented atdisplay 112.

In other examples, display 112 may present cursor 306 on the display oras part of a graphical user interface, and such a cursor 306 may operatein a different manner. In such examples, presence-sensitive panel 102may operate as a trackpad, and cursor 306 may react to input atpresence-sensitive panel 102 in a manner similar to a cursor displayedon a laptop screen having a trackpad for input. In such an example,computing device 200 may control the movement of cursor 306 on display112 based on input detected at presence-sensitive panel 102. Forinstance, presence-sensitive panel 102 may detect one or more inputs,movements, or gestures along the surface of presence-sensitive panel102. Computing device 200 may identify, based on the input, a directionof movement and a distance of movement. Responsive to such adetermination, computing device 200 may move cursor 306 on display 112in the determined direction of movement and for the determined distanceof movement within the graphical user interface presented at display112. In one example, computing device 200 may move cursor 306 so thatcursor 306 overlaps one of user interface elements 304.

FIG. 3D is a conceptual diagram illustrating a perspective view from therear of the interior of a vehicle, including an example vehiclecomputing system 100 capable of providing feedback for touch input inaccordance with one or more aspects of the present disclosure. In FIG.3D, computing device 200 has, in response to input detected atpresence-sensitive panel 102, moved cursor 306 so that cursor 306overlaps one of user interface elements 304. When cursor 306 is over oneof user interface elements 304, presence-sensitive panel 102 may detectone or more taps or inputs at presence-sensitive panel 102, andcomputing device 200 may determine, based on this input, that the inputcorresponds to selection of the user interface elements 304 overlappedby cursor 306. Alternatively, after cursor 306 is moved over one of userinterface elements 304, camera 104 may detect input in the form of oneor more images of an arm or hand within field of view 160, and computingdevice 200 may determine, based on this input detected by camera 104,that the input corresponds to selection of the user interface element304 overlapped by cursor 306. Responsive to detecting inputcorresponding to selection of user interface element 304, computingdevice 200 may perform an operation corresponding to the user interfaceelement 304 overlapped by cursor 306.

In some examples, display 112 may be a presence-sensitive panel thatoperates both as an input device and an output device. In such anexample, display 112 may detect one or more inputs at or near a locationon display 112 where display 112 presents user interface element 304.Computing device 200 may identify, based on the input, a selected userinterface element 304 corresponding to the input. In response to theinput selecting user interface element 304, computing device 200 mayperform an operation, which may include displaying information orupdating a graphical user interface at display 112. In some exampleswhere presence-sensitive panel 102 also acts as a display, computingdevice 200 may additionally or alternatively display information atpresence-sensitive panel 102 or update a graphical user interfacedisplayed at presence-sensitive panel 102.

FIG. 4A through FIG. 4D illustrate one or more example operationsrelating to detecting input by camera 104. For purposes of illustration,FIG. 4A through FIG. 4E are described within the context of vehiclecomputing system 100 of FIG. 1 and FIG. 2.

FIG. 4A is a conceptual diagram illustrating a side view of an interiorof a vehicle, including an example vehicle computing system 100 capableof detecting gesture input in an example field of view 160 in accordancewith one or more aspects of the present disclosure. Seated on seat 152is user 150. As in FIG. 3A through FIG. 3D, user 150 may be a driver,passenger, or other vehicle occupant. User 150 may be seated in thefront seat or at another location within the vehicle.

In the example of FIG. 4A, user 150 may interact with one or morecomponents of the vehicle, and/or provide input at input devices 210 orpresence-sensitive panel 102 or camera 104. In FIG. 4A, user 150 isshown interacting with camera 104.

Camera 104 may detect input including images of one or more movements orgestures made by user 150 within field of view 160. Such movements orgestures may be hand movements, arm movements, or finger gestures withinfield of view 160. In some examples, such movements may be within fieldof view 160 but outside the threshold distance previously described inconnection with presence-sensitive panel 102, where such a thresholddistance may be measured upward from the surface of presence-sensitivepanel 102. Camera 104 may be positioned or configured so that field ofview 160 is located within the vehicle such that user 150 performing amovement or a gesture intended as input for camera 104 might requireuser 150 to lift his or her arm or hand off center console 101 little orno distance. Camera 104 may be positioned or configured such that user150 performing a movement or a gesture within field of view 160 as inputfor camera 104 is less likely to have such a movement or gesture beinterpreted as an attempt to communicate with someone outside thevehicle. In other words, in some examples camera 104 and field of view160 may be positioned low enough within the vehicle so that apedestrian, for example, is less likely to see gestures that user 150 ora driver or vehicle occupant may perform within the vehicle fordetection by camera 104.

Computing device 200 may determine, based on the input detected bycamera 104, a command or operation corresponding to the input. Computingdevice 200 may perform the command or operation, or may cause display112 to display information about the command or operation, or thecommand or operation to be performed. For instance, computing device 200may present a prompt at display 112 relating to the command oroperation.

FIG. 4B is a conceptual diagram illustrating a top view of an examplepresence-sensitive panel 102 and an example camera 104. In the exampleof FIG. 4B, camera 104 is mounted in the manner shown in FIG. 4A, underpresence-sensitive panel 102. Camera 104 may be an infrared camera, andpresence-sensitive panel 102 may be transparent to infrared light, sothat camera 104 is able to capture images above presence-sensitive panel102. An infrared camera may, in some examples, enable effective captureof images during conditions of low levels of ambient light (e.g., atnight), and during conditions of high levels of ambient light (e.g., ina convertible on a sunny day, or when a sunroof provides significantlight through roof 158). An infrared camera may be more appropriate thanother types of cameras, such as an optical camera, in certain varyinglight conditions.

In FIG. 4B, camera 104 may detect one or more images of an arm movement.Computing device 200 may determine, based on the one or more images,that the arm may be moving in the direction indicated in FIG. 4B.Computing device 200 may further determine that such a movementcorresponds to a swiping gesture.

FIG. 4C is a conceptual diagram illustrating a perspective view from therear of the interior of a vehicle, including an example vehiclecomputing system 100 capable of detecting gesture input in accordancewith one or more aspects of the present disclosure. FIG. 4C illustratescenter console 101 with camera 104 positioned on or within centerconsole 101. As in FIG. 4B, presence-sensitive panel 102 is positionedon top of camera 104, but camera 104 may nevertheless capture imagesabove presence-sensitive panel 102 within field of view 160.

In the example of FIG. 4C, computing device 200 has detected an incomingphone call from one or more of communication units 216, and has causeddisplay 112 to present a prompt 310 requesting that user 150 answer ordecline the call. Camera 104 detects input in the form of one or moreimages of an arm or hand within field of view 160. Computing device 200determines, based on the input, a gesture corresponding to the input. Inthis example, if computing device 200 determines that the gesturecorresponds to a “swipe left” gesture, computing device 200 connects oranswers the call, enabling communication between one or more vehicleoccupants and one or more other persons (e.g., Elizabeth in theillustration of FIG. 4B).

FIG. 4D is a conceptual diagram illustrating a perspective view from therear of the interior of a vehicle, including an example vehiclecomputing system 100 capable of providing feedback for gesture input inaccordance with one or more aspects of the present disclosure. In theexample of FIG. 4C, after computing device 200 connects or answers thecall, computing device 200 may then update display 112 to providefeedback or visual confirmation that the call has been connected. SeeFIG. 4D. Again referring to the example of FIG. 4C, if computing device200 determines, based on the input detected by camera 104, that thedetected gesture corresponds to a “swipe right” gesture, computingdevice 200 might not connect the call, thereby disabling communication.In some examples, computing device 200 may in such a situation updatedisplay 112 to provide feedback or visual confirmation that the call hasnot been connected.

Referring again to FIG. 4C, the graphical user interface presented atdisplay 112 includes cursor 306. In some examples, computing device 200may control the movement of cursor 306 in response to input detected bycamera 104, in a manner analogous to the manner in which computingdevice 200 controlled the movement of cursor 306 in response to inputdetected by presence-sensitive panel 102 in FIG. 3D, as previouslydescribed. For instance, camera 104 may detect input in the form of oneor more images of an arm or hand within field of view 160. Computingdevice 200 may identify, based on the input, a direction of movement anda distance of movement. Responsive to such a determination, computingdevice 200 may move cursor 306 in the determined direction of movementand for the determined distance of movement within the graphical userinterface presented at display 112. In one example, computing device 200may move cursor 306 so that cursor 306 overlaps one of user interfaceelements 304.

After computing device 200 moves cursor 306 over one of user interfaceelements 304, camera 104 may detect input in the form of one or moreimages of an arm or hand within field of view 160, and computing device200 may determine, based on this input at camera 104, that the inputcorresponds to selection of user interface element 304 overlapped bycursor 306. Alternatively, after cursor 306 is moved over one of userinterface elements 304, presence-sensitive panel 102 may detect one ormore taps or inputs at presence-sensitive panel 102, and computingdevice 200 may determine, based on this input at presence-sensitivepanel 102, that the input corresponds to selection of user interfaceelement 304 overlapped by cursor 306. Responsive to detecting inputcorresponding to selection of user interface element 304, computingdevice 200 may perform an operation corresponding to the user interfaceelement 304 overlapped by cursor 306.

FIG. 4E is a conceptual diagram illustrating a perspective view from therear of the interior of a vehicle, including an example vehiclecomputing system 100 capable of detecting another type of gesture inputin accordance with one or more aspects of the present disclosure. As inFIG. 4C and FIG. 4D, FIG. 4E illustrates center console 101 with camera104 positioned on or within center console 101, with presence-sensitivepanel 102 positioned on top of camera 104. Camera 104 may be an infraredcamera, and presence-sensitive panel 102 may be transparent to infraredlight.

In the example of FIG. 4E, computing device 200 is presenting aninterface at display 112 for controlling cabin temperature. Camera 104detects input in the form of one or more images of an arm or hand withinfield of view 160. Computing device 200 determines, based on the input,a gesture corresponding to the input. In this example, computing device200 may determine that the gesture corresponds to fingers moving apart,which computing device 200 may interpret as an “increase” gesture.Computing device 200 may therefore interpret such a gesture as a commandto increase the cabin temperature, and computing device 200 may increasea setting corresponding to the cabin temperature.

Computing device 200 may be configured to interpret input detected bycamera 104 as a variety of gestures. Such gestures may include, but arenot limited to, a swiping motion (e.g., as illustrated in FIG. 4B), apinching motion of two fingers (e.g., as illustrated in FIG. 4E), agrasping and/or turning gesture (e.g., as might be used to remove a capfrom a twist-off bottle), a knob-turning gesture, an upwards ordownwards gesture, a movement gesture (e.g., from left to right, rightto left, up to down, down to up, front to back, back to front), apushing gesture, opening or closing a fist, a tapping gesture, a wavinggesture, an opening or closing gesture, a clapping gesture, an open handor open palm gesture, a splayed fingers gesture, a pointing gesture, andturning or reversing gesture, a clockwise or counterclockwise gesture,or a sliding, drag, or release gesture.

FIG. 5 is a flow diagram illustrating example operations of an examplevehicle computing system 100 that implements techniques for recognizingmultiple types of input, in accordance with one or more aspects of thepresent disclosure. The process of FIG. 5 may be performed by one ormore processors of a computing device, such as computing device 200 ofFIG. 2. For purposes of illustration, FIG. 5 is described within thecontext of vehicle computing system 100 and computing device 200 of FIG.1 and FIG. 2.

In the example of FIG. 5, computing device 200 may detect input frompresence-sensitive panel 102, camera 104, or input devices 210 (504).For instance, presence-sensitive panel 102 may detect one or more taps,gestures, and/or other user inputs at locations of presence-sensitivepanel 102, and/or camera 104 may detect images of one or more movementsor gestures made within field of view 160. Computing device 200 maydetermine, based on the input, an operation corresponding to the input(506).

In some examples, computing device 200 may determine a gesture oroperation based only on input detected by presence-sensitive panel 102even though camera 104 may also detect input. For example,presence-sensitive panel 102 may detect a tap gesture (e.g., as a usermoves a finger or stylus within a threshold distance ofpresence-sensitive panel 102) that computing device 200 determinescorresponds to an input for adjusting seat position settings. In such anexample, camera 104 may also detect the same tap gesture as the usermoves the finger or stylus because the movement is within field of view160. Despite camera 104 detecting the tap gesture, computing device 200may determine that since presence-sensitive panel 102 detected the tapgesture, the input detected by camera 104 is duplicative, redundant, orotherwise not necessary and therefore should be disregarded ordiscarded.

In other examples, computing device 200 may determine a gesture oroperation based only on input detected by camera 104 even thoughpresence-sensitive panel 102 may also detect input. For example,presence-sensitive panel 102 may detect a hand gesture (e.g., as a usermoves a hand beyond a threshold distance of presence-sensitive panel102) within field of view 160 that computing device 200 determinescorresponds to an input for tuning a radio to the next preset radiostation. In such an example, presence-sensitive panel 102 may alsodetect aspects of the same hand gesture (e.g., if the hand gesture isnear the threshold distance that computing device 200 may use to discerntwo-dimensional versus three-dimensional gestures). Even thoughpresence-sensitive panel 102 may detect aspects of the same handgesture, computing device 200 may determine that the input detected bycamera 104 should be considered stronger or more significant, andtherefore the input detected by presence-sensitive panel 102 isduplicative, redundant, or otherwise not necessary and therefore shouldbe disregarded or discarded. In other words, even thoughpresence-sensitive panel 102 may detect movements within field of view160 if such movements are close enough to presence-sensitive panel 102to be within the threshold distance of the surface of presence-sensitivepanel 102, computing device 200 may disregard such movements if camera104 also detects movements simultaneously occurring beyond the thresholddistance of the surface of presence-sensitive panel 102.

In other examples, computing device 200 may determine a gesture oroperation based on input detected by both presence-sensitive panel 102and camera 104. In some examples, presence-sensitive panel 102 andcamera 104 may detect input independently and/or simultaneously, andcomputing device 200 may determine, based on input detected by bothdevices, a gesture or operation. Computing device 200 may use inputdetected by presence-sensitive panel 102 to help confirm a gesture oroperation determined based on input detected by camera 104, or computingdevice 200 may use input detected by camera 104 to help confirm agesture or operation determined based on input detected bypresence-sensitive panel 102. For example, computing device 200 maydetermine that input detected by presence-sensitive panel 102 likelycorresponds to a particular tap gesture, and computing device 200 mayevaluate input detected by camera 104 to ensure that such input isconsistent with that tap gesture. For instance, computing device 200 mayconfirm that a hand was detected above presence-sensitive panel 102 infield-of-view 160, as would be expected for a tap gesture performed by afinger. If not, computing device 200 may determine that the inputdetected by presence-sensitive panel 102 was a different gesture or afalse input.

In other examples, computing device 200 may determine a gesture,operation, or command based on input detected by both presence-sensitivepanel 102 and camera 104 that might not be detectable (or as easilydetectable) through input from only presence-sensitive panel 102 orinput from only from camera 104. In other words, in such an example,computing device 200 may determine a gesture, operation, or commandbased on the combined input from both presence-sensitive panel 102 andcamera 104. For example, one or more gestures could involve movements inboth three-dimensional space within field of view 160 along with, orfollowed by, a movement in the approximately two-dimensional space onthe surface of presence-sensitive panel 102. For instance, such agesture may involve a hand or arm movement within field-of-view 160ending in contact with the presence-sensitive panel 102.

In further examples, computing device 200 may determine an operationbased on input from some combination of input detected bypresence-sensitive panel 102, camera 104, and/or input devices 210. Insuch examples, computing device 200 may use input detected by inputdevices 210 to confirm input or a gesture or operation detected bypresence-sensitive panel 102 and/or camera 104. In other examples,computing device 200 may use input detected by input devices 210, incombination with input detected by presence-sensitive panel 102 andcamera 104, to determine a gesture, operation, or command based on thecombined input.

In response to determining an operation, computing device 200 mayprovide feedback based on the input (508). Such feedback may be visual,or audio, or in another form. In some examples, computing device 200 maycause feedback to be presented at display 112. For instance, ifcomputing device 200 determines that the operation corresponding toinput is a request to provide guidance to a home address, computingdevice 200 may cause display 112 to display a prompt requestingconfirmation that such an operation be initiated. In some examples,feedback may not be provided for some or all operations, and in suchexamples, providing feedback (508) may be optional.

Presence-sensitive panel 102, camera 104, and/or input devices 210 maydetect further input confirming or declining the operation (510). If theoperation is confirmed, computing device 200 may perform the operation(512). In some examples, after computing device 200 determines anoperation to be performed, a confirmation step might not be performed,so confirming the operation (510) may be optional. For some operations,it may be more efficient to simply perform the operation without firstrequiring confirmation. In some examples, feedback may nevertheless beprovided, even if confirmation is not required. For instance, ifcomputing device 200 determines that the operation corresponding todetected input is a request to mute an audio system, computing device200 may, in some examples, perform the muting operation, and may causedisplay 112 to present a notice that the audio system was muted inresponse to a user command.

FIG. 6 is a flow diagram illustrating example operations of an examplevehicle computing system 100 that implements techniques for recognizingmultiple types of input, in accordance with one or more aspects of thepresent disclosure. The operations of FIG. 6 may be performed by one ormore processors of a computing device, such as computing device 200 ofFIG. 2. For purposes of illustration, FIG. 6 is described within thecontext of vehicle computing system 100 and computing device 200 of FIG.1 and FIG. 2.

In operation, computing device 200 may receive, from presence-sensitivepanel 102, a first indication of input (602). For example, computingdevice 200 may detect one or taps or gestures at presence-sensitivepanel 102 performed by the driver of the vehicle.

Computing device 200 may receive, from camera 104, a second indicationof input (604). For example, camera 104 may detect one or more images ofone or more movements or gestures made by a driver within field of view160.

Computing device 200 may determine, based on at least one of the firstindication of input or the second indication of input, an operation tobe performed. For example, computing device 200 may determine that theinput from either presence-sensitive panel 102, or camera 104 (or both)correspond to a particular operation, such as starting a navigationapplication.

Clause 1. A computing system of a vehicle comprising: apresence-sensitive panel within vehicle; an infrared camera configuredto capture images in a three-dimensional space within the vehicle; atleast one processor; and at least one storage device that storesinstructions that, when executed, cause the at least one processor to:receive, from the presence-sensitive panel, a first indication of input,receive, from the infrared camera, a second indication of input, anddetermine, based on at least one of the first indication of input or thesecond indication of input, an operation to be performed.

Clause 2. The computing system of clause 1, wherein thepresence-sensitive panel is positioned between the infrared camera andthe three-dimensional space and is transparent to infrared; and whereinthe infrared camera is positioned under the presence-sensitive panel andis oriented to capture images in the three-dimensional space above thepresence-sensitive panel.

Clause 3. The computing system of clause 2, wherein thepresence-sensitive panel is positioned substantially horizontally on acentral console of the vehicle.

Clause 4. The computing system of any of clauses 1-3, wherein thepresence-sensitive panel is configured to detect two-dimensional gestureinputs performed within a threshold distance of the presence-sensitivepanel; and wherein the infrared camera is configured to detectthree-dimensional gesture inputs performed beyond the threshold distanceof the input surface of the presence-sensitive panel and in thethree-dimensional space.

Clause 5. The computing system of clause any of clauses 1-4, whereindetermining the operation to be performed is based on both the firstindication of input and the second indication of input.

Clause 6. The computing system of any of clauses 1-5, whereindetermining the operation to be performed is based on the firstindication of input.

Clause 7. The computing system of clause 6, wherein the operation to beperformed is a first operation to be performed, and wherein theinstructions, when executed, further cause the at least one processorto: responsive to determining the first operation to be performed,output, for display, information about the first operation to beperformed; determine, based on the second indication of input, a secondoperation to be performed; and responsive to determining the secondoperation to be performed, output, for display, information about thesecond operation to be performed.

Clause 8. The computing system of clause 6, further comprising a displayscreen, wherein the operation to be performed is a first operation to beperformed, and wherein the instructions, when executed, further causethe at least one processor to: responsive to determining the firstoperation to be performed, display on the display screen informationabout the first operation to be performed; determine, based on thesecond indication of input, a second operation to be performed; andresponsive to determining the second operation to be performed, displayon the display screen information about the second operation to beperformed.

Clause 9. The computing system of clause 6, further comprising a displayscreen, and wherein the instructions, when executed, further cause theat least one processor to: responsive to determining the operation to beperformed and prior to receiving the second indication of input, displayon the display screen information requesting confirmation to perform theoperation; determine, based on the second indication of input, that theoperation should be performed; and perform the operation.

Clause 10. The computing system of clause 6, wherein thethree-dimensional space has high ambient light conditions, and whereindetermining the operation to be performed is based on the secondindication of input and includes determining gestures from imagescaptured by the infrared camera in the high ambient light conditions.

Clause 11. The computing system of clause 6, wherein thethree-dimensional space has low ambient light conditions, and whereindetermining the operation to be performed is based on the secondindication of input and includes determining gestures from imagescaptured by the infrared camera in the low ambient light conditions.

Clause 12. The computing system of clause 6, wherein thethree-dimensional space includes structure in a field of view of theinfrared camera, and wherein determining the operation to be performedis based on the second indication of input and includes discerninggestures from the structure in the field of view.

Clause 13. A method comprising: receiving, by a computing system of avehicle, from a presence-sensitive panel positioned within the vehicle,a first indication of input; receiving, by the computing system, from aninfrared camera configured to capture images in a three-dimensionalspace within the vehicle, a second indication of input; and determining,based on at least one of the first indication of input or the secondindication of input, an operation to be performed.

Clause 14. The method of clause 13, wherein the presence-sensitive panelis positioned between the infrared camera and the three-dimensionalspace and is transparent to infrared; and wherein the infrared camera ispositioned under the presence-sensitive panel and is oriented to captureimages in the three-dimensional space above the presence-sensitivepanel.

Clause 15. The method of clause 14, wherein the presence-sensitive panelis positioned substantially horizontally on a central console of thevehicle.

Clause 16. The method of any of clauses 13-15, wherein thepresence-sensitive panel is configured to detect two-dimensional gestureinputs performed within a threshold distance of the presence-sensitivepanel; and wherein the infrared camera is configured to detectthree-dimensional gesture inputs performed beyond the threshold distanceof the input surface of the presence-sensitive panel and in thethree-dimensional space.

Clause 17. The method of any of clauses 13-16, wherein determining theoperation to be performed is based on both the first indication of inputand the second indication of input.

Clause 18. The method of any of clauses 13-17, wherein determining theoperation to be performed is based on the first indication of input.

Clause 19. The method of clause 18, wherein the operation to beperformed is a first operation to be performed, the method furthercomprising: responsive to determining the first operation to beperformed, outputting, for display, information about the firstoperation to be performed; determining, based on the second indicationof input, a second operation to be performed; and responsive todetermining the second operation to be performed, outputting, fordisplay, information about the second operation to be performed.

Clause 20. A computer-readable storage medium comprising instructionsthat, when executed by a computing system of a vehicle, cause at leastone processor of the computing system to: receive, from apresence-sensitive panel positioned within the vehicle, a firstindication of input; receive, from an infrared camera configured tocapture images in a three-dimensional space within the vehicle, a secondindication of input; and determine, based on at least one of the firstindication of input or the second indication of input, an operation tobe performed.

Clause 21. A computing system comprising means for performing any of themethods of clauses 13-19.

In one or more examples, the functions described may be implemented inhardware, software, firmware, or any combination thereof. If implementedin software, the functions may be stored on or transmitted over, as oneor more instructions or code, a computer-readable medium and executed bya hardware-based processing unit. Computer-readable media may includecomputer-readable storage media, which corresponds to a tangible mediumsuch as data storage media, or communication media including any mediumthat facilitates transfer of a computer program from one place toanother, e.g., according to a communication protocol. In this manner,computer-readable media generally may correspond to (1) tangiblecomputer-readable storage media, which is non-transitory or (2) acommunication medium such as a signal or carrier wave. Data storagemedia may be any available media that can be accessed by one or morecomputers or one or more processors to retrieve instructions, codeand/or data structures for implementation of the techniques described inthis disclosure. A computer program product may include acomputer-readable medium.

By way of example, and not limitation, such computer-readable storagemedia can comprise RAM, ROM, EEPROM, CD-ROM or other optical diskstorage, magnetic disk storage, or other magnetic storage devices, flashmemory, or any other medium that can be used to store desired programcode in the form of instructions or data structures and that can beaccessed by a computer. Also, any connection is properly termed acomputer-readable medium. For example, if instructions are transmittedfrom a website, server, or other remote source using a coaxial cable,fiber optic cable, twisted pair, digital subscriber line (DSL), orwireless technologies such as infrared, radio, and microwave, then thecoaxial cable, fiber optic cable, twisted pair, DSL, or wirelesstechnologies such as infrared, radio, and microwave are included in thedefinition of medium. It should be understood, however, thatcomputer-readable storage media and data storage media do not includeconnections, carrier waves, signals, or other transient media, but areinstead directed to non-transient, tangible storage media. Disk anddisc, as used, includes compact disc (CD), laser disc, optical disc,digital versatile disc (DVD), floppy disk and Blu-ray disc, where disksusually reproduce data magnetically, while discs reproduce dataoptically with lasers. Combinations of the above should also be includedwithin the scope of computer-readable media.

Instructions may be executed by one or more processors, such as one ormore digital signal processors (DSPs), general purpose microprocessors,application specific integrated circuits (ASICs), field programmablelogic arrays (FPGAs), or other equivalent integrated or discrete logiccircuitry. Accordingly, the term “processor,” as used may refer to anyof the foregoing structure or any other structure suitable forimplementation of the techniques described. In addition, in someaspects, the functionality described may be provided within dedicatedhardware and/or software modules. Also, the techniques could be fullyimplemented in one or more circuits or logic elements.

The techniques of this disclosure may be implemented in a wide varietyof devices or apparatuses, including a wireless handset, an integratedcircuit (IC) or a set of ICs (e.g., a chip set). Various components,modules, or units are described in this disclosure to emphasizefunctional aspects of devices configured to perform the disclosedtechniques, but do not necessarily require realization by differenthardware units. Rather, as described above, various units may becombined in a hardware unit or provided by a collection ofinteroperative hardware units, including one or more processors asdescribed above, in conjunction with suitable software and/or firmware.

Various examples have been described. These and other examples may bewithin the scope of one or more of the following claims.

What is claimed is:
 1. A computing system of a vehicle comprising: apresence-sensitive panel within the vehicle; an infrared cameraconfigured to capture images in a three-dimensional space within thevehicle; at least one processor; and at least one storage device thatstores instructions that, when executed, cause the at least oneprocessor to: receive, from the presence-sensitive panel, a firstindication of input, receive, from the infrared camera, a secondindication of input, and determine, based on at least one of the firstindication of input or the second indication of input, an operation tobe performed.
 2. The computing system of claim 1, wherein thepresence-sensitive panel is positioned between the infrared camera andthe three-dimensional space and is transparent to infrared; and whereinthe infrared camera is positioned under the presence-sensitive panel andis oriented to capture images in the three-dimensional space above thepresence-sensitive panel.
 3. The computing system of claim 2, whereinthe presence-sensitive panel is positioned substantially horizontally ona central console of the vehicle.
 4. The computing system of claim 1,wherein the presence-sensitive panel is configured to detecttwo-dimensional gesture inputs performed within a threshold distance ofthe presence-sensitive panel; and wherein the infrared camera isconfigured to detect three-dimensional gesture inputs performed beyondthe threshold distance of an input surface of the presence-sensitivepanel and in the three-dimensional space.
 5. The computing system ofclaim 1, wherein determining the operation to be performed is based onboth the first indication of input and the second indication of input.6. The computing system of claim 1, wherein determining the operation tobe performed is based on the first indication of input.
 7. The computingsystem of claim 6, wherein the operation to be performed is a firstoperation to be performed, and wherein the instructions, when executed,further cause the at least one processor to: responsive to determiningthe first operation to be performed, output, for display, informationabout the first operation to be performed; determine, based on thesecond indication of input, a second operation to be performed; andresponsive to determining the second operation to be performed, output,for display, information about the second operation to be performed. 8.The computing system of claim 6, further comprising a display screen,wherein the operation to be performed is a first operation to beperformed, and wherein the instructions, when executed, further causethe at least one processor to: responsive to determining the firstoperation to be performed, display on the display screen informationabout the first operation to be performed; determine, based on thesecond indication of input, a second operation to be performed; andresponsive to determining the second operation to be performed, displayon the display screen information about the second operation to beperformed.
 9. The computing system of claim 6, further comprising adisplay screen, and wherein the instructions, when executed, furthercause the at least one processor to: responsive to determining theoperation to be performed and prior to receiving the second indicationof input, display on the display screen information requestingconfirmation to perform the operation; determine, based on the secondindication of input, that the operation should be performed; and performthe operation.
 10. The computing system of claim 6, wherein thethree-dimensional space has high ambient light conditions, and whereindetermining the operation to be performed is based on the secondindication of input and includes determining gestures from imagescaptured by the infrared camera in the high ambient light conditions.11. The computing system of claim 6, wherein the three-dimensional spacehas low ambient light conditions, and wherein determining the operationto be performed is based on the second indication of input and includesdetermining gestures from images captured by the infrared camera in thelow ambient light conditions.
 12. The computing system of claim 6,wherein the three-dimensional space includes structure in a field ofview of the infrared camera, and wherein determining the operation to beperformed is based on the second indication of input and includesdiscerning gestures from the structure in the field of view.
 13. Amethod comprising: receiving, by a computing system of a vehicle, from apresence-sensitive panel positioned within the vehicle, a firstindication of input; receiving, by the computing system, from aninfrared camera configured to capture images in a three-dimensionalspace within the vehicle, a second indication of input; and determining,based on at least one of the first indication of input or the secondindication of input, an operation to be performed.
 14. The method ofclaim 13, wherein the presence-sensitive panel is positioned between theinfrared camera and the three-dimensional space and is transparent toinfrared; and wherein the infrared camera is positioned under thepresence-sensitive panel and is oriented to capture images in thethree-dimensional space above the presence-sensitive panel.
 15. Themethod of claim 14, wherein the presence-sensitive panel is positionedsubstantially horizontally on a central console of the vehicle.
 16. Themethod of claim 13, wherein the presence-sensitive panel is configuredto detect two-dimensional gesture inputs performed within a thresholddistance of the presence-sensitive panel; and wherein the infraredcamera is configured to detect three-dimensional gesture inputsperformed beyond the threshold distance of an input surface of thepresence-sensitive panel and in the three-dimensional space.
 17. Themethod of claim 13, wherein determining the operation to be performed isbased on both the first indication of input and the second indication ofinput.
 18. The method of claim 13, wherein determining the operation tobe performed is based on the first indication of input.
 19. The methodof claim 18, wherein the operation to be performed is a first operationto be performed, the method further comprising: responsive todetermining the first operation to be performed, outputting, fordisplay, information about the first operation to be performed;determining, based on the second indication of input, a second operationto be performed; and responsive to determining the second operation tobe performed, outputting, for display, information about the secondoperation to be performed.
 20. A computer-readable storage mediumcomprising instructions that, when executed by a computing system of avehicle, cause at least one processor of the computing system to:receive, from a presence-sensitive panel positioned within the vehicle,a first indication of input; receive, from an infrared camera configuredto capture images in a three-dimensional space within the vehicle, asecond indication of input; and determine, based on at least one of thefirst indication of input or the second indication of input, anoperation to be performed.